Agricultural Machine Guidance

ABSTRACT

Systems and methods are provided for guiding an agricultural machine utilising image data received from one or more imaging sensors associated with the agricultural machine. The image data is analysed utilising a detection model to classify at least one object within the environment of the agricultural machine and the classification is used to identify a cooperative machine for performing a cooperative operational task with the agricultural machine. The systems extend to controlling operation of the guidance system in dependence on the identified cooperative machine.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to United Kingdom patent applicationnumber GB 2208929.6, filed Jun. 17, 2022, the entirety of which ishereby incorporated by reference.

FIELD

Embodiments of the present disclosure relate generally to systems andmethods for controlling a guidance system for an agricultural machine,and specifically for use with a cooperative machine for performing acooperative operational task.

BACKGROUND

Many agricultural operational tasks involve the use of multipleagricultural machines operating cooperatively. As an example, a graincart or trailer may be driven alongside a harvesting machine whilstharvesting to enable simultaneous unloading of the harvested cropmaterial from the harvesting machine to the cart/trailer. This allowsthe harvesting machine to continue the harvesting task withoutcompletely filling its grain bin and thereby requiring a stop in theharvesting task to perform a separate unloading step. This may extend tomultiple like machines operating in parallel across a workingenvironment to improve the efficiency (and ultimately the productivity)of a given task.

To further increase the efficiency of such tasks, some systems arebeginning to provide the ability to automate certain tasks. This mayinclude automating the steering and/or drive of a machine, for example.When applied to cooperative tasks this may include automating themovement of a “follower” machine with respect to a “leader” machine (orvice versa). In the example given above, movement of the graincart/trailer may be automated (at least partly) to synchronize itsoperation with the harvesting machine. Such systems can be rathercomplex and often require a stable communication link between the leaderand follower vehicles, and/or a remote operational sever controlling theoperation. As will be appreciated, in an agricultural context in remoteand varying conditions this may not always be possible, and hence suchknown systems may only be useable in certain operating circumstances.

It would be advantageous to provide a system which overcomes or at leastpartly addresses problems associated with known systems.

BRIEF SUMMARY

In an aspect of the invention there is provided a control system for aguidance system of an agricultural machine, the control systemcomprising one or more controllers, and being configured to: receiveimage data from one or more imaging sensors associated with theagricultural machine; analyse the image data utilising a detection modelto classify at least one object within the environment of theagricultural machine; identify, from the at least one classified object,a cooperative machine for performing a cooperative operational task withthe agricultural machine; and generate and output one or more controlsignals for controlling operation of the guidance system in dependenceon the identified cooperative machine.

Advantageously, the present invention utilises a detection model foridentifying from image data a cooperative machine within the environmentof the agricultural machine and controls a guidance system of theagricultural machine accordingly, e.g. for performing a cooperativeoperational task. Such as system may provide an onboard vision systemfor automating at least part of a cooperative operational task whichdoes not necessarily rely on a data communication between thecooperative machine and the agricultural machine, and as such may beworkable in many different operating conditions.

The one or more controllers may collectively comprise an input (e.g. anelectronic input) for receiving one or more input signals. The one ormore input signals may comprise image data from the one or more imagingsensors. The one or more controllers may collectively comprise one ormore processors (e.g. electronic processors) operable to executecomputer readable instructions for controlling operational of thecontrol system, for example, to analyse the image data and/or toidentify the cooperative machine. The one or more processors may beoperable to generate one or more control signals for controllingoperation of the guidance system. The one or more controllers maycollectively comprise an output (e.g. an electronic output) foroutputting the one or more control signals.

The control system may be operable to control the guidance system independence on the identity of the identified cooperative machine. Thecontrol system may be operable to control the guidance system independence on the position of the identified cooperative machine.

The guidance system may comprise a user interface. The control systemmay be configured to control operation of the user interface foroutputting one or more indicators indicative of the identity and/orposition of the cooperative machine. This may include highlighting, e.g.to an operator of the agricultural machine, the position of thecooperative machine with respect to the agricultural machine. This mayinclude presenting on a graphical user interface an image, e.g. usingimage data from the imaging sensor(s) and/or a virtual image,highlighting the position of the cooperative machine.

The control system may be configured to control operation of the userinterface of the guidance system for outputting one or more instructionsfor an operator of the agricultural machine relating to movement of theagricultural machine with respect to the cooperative machine. This mayinclude suggestion of a speed and/or operational path for theagricultural machine for guiding the machine with respect to thecooperative machine, e.g. for performing the cooperative operationaltask.

The user interface may comprise a display terminal of the machine, forexample, a display terminal provided within an operator cab of theagricultural machine, or a remote user device, which may be operable incombination with the agricultural machine.

The control system may be configured to determine an operational pathfor the agricultural machine for performing the cooperative operationaltask. The operational path may comprise a direction and/or speed for theagricultural machine.

The control system may be configured to generate and output one or morecontrol signals for controlling operation of the guidance system of theagricultural machine for controlling motion of the agricultural machinealong the determined operational path. This may include control over apropulsion system of the agricultural machine for controlling a forwardspeed of the agricultural machine. The forward speed of the machine maycomprise an absolute speed of the machine, or a relative speed withrespect to the cooperative machine. The speed of the cooperative machinemay be determined utilising sensor data, which may include image datafrom the imaging sensor(s) for example. This may include utilisingsensor data from a LIDAR sensor monitoring a change in distance betweenthe LIDAR sensor (and hence the agricultural machine) and thecooperative machine.

The control system may be configured to control a steering system of theagricultural machine, e.g. for controlling steering of the machine alongor with respect to the determined operational path, or with respect tothe cooperative machine or one or more components thereof. Here, themovement of the cooperative machine may be determined utilising sensordata, which may include image data from the imaging sensors. This mayinclude monitoring the relative position of the cooperative machine, orone or more components thereof (e.g. an unloading auger) utilisingsensor data from a LIDAR sensor providing depth information between theLIDAR sensor (and hence the agricultural machine or an appropriatecomponent thereof—e.g. a trailer) and the cooperative machine (or theappropriate component thereof, e.g. an unloading auger).

The operational path may comprise a suggested or determined path formoving the agricultural machine to an operational position with respectto the cooperative machine. This may include an operational path betweenthe current position of the agricultural machine and a position, e.g.with respect to the cooperative machine, suitable for performing thecooperative operational task.

The operational path may comprise a suggested or determined path formoving the agricultural machine along a path to perform the cooperativeoperational task.

In some scenarios it may be possible for multiple objects to be presentin the environment of the agricultural machine. The control system maybe configured to utilise the detection model for determining aclassification for one or more of a plurality of objects within theimage data. The control system may be configured to determine, from themultiple classifications, one or more candidate cooperative machines.The control system may be configured to select one of the one or morecandidate cooperative machines and control the guidance system of theagricultural machine in dependence thereon. This may include selectionof the candidate machine which is closest in proximity to theagricultural machine, for example.

The one or more imaging sensors may include a camera. The one or moreimaging sensors may comprise a LIDAR sensor. Where used, a LIDAR sensormay be used to obtain information relating to the distance between theobject and the sensor, and as such can be used in conjunction with theimage data from another sensor, e.g. a camera, for classification by thedetection model, for example, for analysing the expected size of a givenobject within the camera data using the depth information from the LIDARsensor.

The detection model may comprise a machine-learned model. Themachine-learned model may be trained on one or more training datasetswith known objects with respective classifications. The machine-learnedmodel may comprise a deep learning model utilising an object detectionalgorithm. The deep learning model may include a YOLO detectionalgorithm, such as a YOLOv5 detection model, for example. The trainingdataset(s) for the model may comprise an agricultural dataset,comprising training images including agricultural-specific objects.

Classification by the detection model may comprise assignment of a classto the object. The class may be one of a plurality of classes for therespective model, as determined during the learning process throughassignment of suitable labels to known objects. The plurality of classesmay be grouped by category, and optionally by subcategory. For example,the plurality of classes may include ‘tractor’, ‘combine’, ‘car’,‘truck’, ‘trailer’, ‘baler’, ‘combine header’, ‘square bale’, ‘roundbale’, ‘person’, and ‘animal’, for example. The classes may be groupedinto categories, which may include, for example, ‘Vehicle’, ‘Implement’,‘Static Object’, and ‘Dynamic Object’. The ‘Vehicle’ category may befurther divided into sub-categories, including ‘Agricultural’—forexample including classes ‘tractor’ and ‘combine’—and ‘Road’—includingclasses ‘car’, and ‘truck’, for example. The ‘Implement’ category may befurther divided into sub-categories, including “Tractor’—for exampleincluding classes ‘trailer’ and ‘baler’—and ‘Combine’—including the‘combine header’ class, for example. The ‘Static Object’ category mayinclude a sub-category ‘Bale’, for example including classes ‘roundbale’ and ‘square bale’. The ‘Dynamic Object’ category may include asub-category ‘Person’, for example including the ‘person’ class.

The classification output by the detection model may comprise a boundingbox overlaid onto the image data at the location of the object asdetermined by the respective model. The position of the bounding boxwithin the image data may be utilised by the control system forcontrolling the guidance system. For example, the position of thebounding box within the image data may be used as an indication of therelative position of the identified cooperative machine with respect tothe agricultural machine.

In an example, the agricultural machine may comprise a tractor with atrailer and the control system may be configured to identify acooperative agricultural machine within the environment of the tractor,e.g. a combine harvester performing a harvesting operation. The controlsystem may be configured to control the guidance system in accordancewith an operational path which aligns the trailer with an unloadingmechanism of the combine for receiving crop material from the combine inthe trailer of the tractor-trailer combination. The present invention isadvantageously used here for positively identifying the combine,identifying the position of the combine with respect to thetractor/trailer combination, and for controlling the guidance system foraligning the tractor/trailer combination with an unloading mechanism,e.g. an unloading auger, of the combine harvester. As discussed herein,control of the guidance system may include automation of the movement ofthe tractor to align the trailer with the harvester, or in someinstances output of instructions to an operator of the tractor.

In a further example, the agricultural machine may comprise a tractorwith an implement suitably coupled thereto. This may be a front mountedimplement or a towed implement coupled to a hitch point of the tractor.In such embodiments, the control system may be configured to identify acooperative agricultural machine within the environment as a tractorwith a corresponding implement coupled thereto. This may be anequivalent implement, for example. The control system may be configuredto control the guidance system in accordance with an operational pathwhich aligns the tractor and implement combination with thecorresponding cooperative machine such that the tractor and implementcombination works an adjacent row to the cooperative machine. Workingtwo (or more) adjacent rows in this fashion may reduce the time taken towork the whole working environment. Again, control of the guidancesystem may include automation of the movement of the tractor to alignthe tractor and implement combination with the cooperative machine, orin some instances output of instructions to an operator of the tractorfor doing so. The implement may, in some instances, comprise a harrow,although the invention is equally applicable to other workingimplements.

In a further example, the agricultural machine may comprise a baler,which may be self-propelled or may be suitably coupled and towed by atractor and the control system may be configured to identify acooperative agricultural machine within the environment of the tractor,e.g. a combine harvester performing a harvesting operation. The controlsystem may be configured to control the guidance system in accordancewith an operational path which aligns the intake of the baler with aswath of residue material spread by the combine during the harvestingoperation. This may include positioning the baler (or tractor/balercombination) directly behind and spaced from the combine. Again, controlof the guidance system may include automation of the movement of thebaler (or tractor/baler combination) to align the baler with theharvester, or in some instances output of instructions to an operator ofthe tractor.

In a further example, the agricultural machine may comprise a harvestingmachine, such as a combine harvester or forage harvester. In suchembodiments, the control system may be configured to identify acooperative agricultural machine within the environment as acorresponding harvesting machine. The control system may be configuredto control the guidance system in accordance with an operational pathwhich aligns the harvesting machine with the corresponding cooperativeharvesting machine such that the harvesting machine works in parallelwith the cooperative harvesting machine—e.g. in the next crop row.Working two (or more) adjacent rows in this fashion may reduce the timetaken to work the whole working environment. Again, control of theguidance system may include automation of the movement of the harvestingmachine with the cooperative harvesting machine, or in some instancesoutput of instructions to an operator of the harvesting machine fordoing so.

In further examples, the cooperative task may include machines workingin different locations in the environment, for example working fromopposite ends of the environment.

In further embodiments the control system may be configured tocommunicate with a data server associated with the agricultural machine.This may comprise a local server or a remote server, for example,accessible via a wireless data communication channel such as a datanetwork, cellular network or the like. The control system may beoperable to store on the data server information indicative of thedetermined cooperative machine. This may be advantageous, for example,where the present invention is utilised to detect and track thepositions of multiple agricultural machines within a workingenvironment. The control system can store this information on the dataserver which in turn may allow, e.g. a remote operator, to remotelytrack a farming operation in the working environment utilising multiplevehicles. Using an image based system on a machine with an appropriatedata communication capability may allow for tracking of multiplemachines and implements within an environment without such systems, orwithout onboard positioning systems such as a GNSS system, and possiblywithout a direct connection between the cooperative machine and theagricultural machine.

A further aspect of the invention comprises a guidance system for anagricultural machine, comprising one or more imaging sensors; whereinthe system further comprises and/or is controllable by the controlsystem of any preceding aspect.

Another aspect of the invention provides an agricultural machinecomprising a control system and/or a guidance system as describedherein.

The agricultural machine may comprise a harvesting machine, such as acombine harvester or forage harvester, for example. The agriculturalmachine may comprise a tractor.

A further aspect of the invention provides a method for guiding anagricultural machine, comprising: receiving image data from one or moreimaging sensors associated with the agricultural machine; analysing theimage data utilising a detection model to classify at least one objectwithin the environment of the agricultural machine; identifying, fromthe at least one classified object, a cooperative machine for performinga cooperative operational task with the agricultural machine; andcontrolling operation of the guidance system in dependence on theidentified cooperative machine.

The method of the present aspect of the invention may compriseperformance of any one or more of the functional features of the controlsystem of a preceding aspect discussed herein.

A further aspect of the invention comprises computer software which,when executed by one or more processors, causes performance of themethod of the preceding aspect of the invention.

An aspect of the invention provides a computer readable storage mediumcomprising the computer software of the preceding aspect of theinvention.

Within the scope of this application it should be understood that thevarious aspects, embodiments, examples and alternatives set out herein,and individual features thereof may be taken independently or in anypossible and compatible combination. Where features are described withreference to a single aspect or embodiment, it should be understood thatsuch features are applicable to all aspects and embodiments unlessotherwise stated or where such features are incompatible.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the invention/disclosure will now bedescribed, by way of example only, with reference to the accompanyingdrawings, in which:

FIG. 1 is a simplified perspective view illustrating an embodiment of anagricultural machine according to the present disclosure;

FIG. 2 is a schematic diagram illustrating an embodiment of a controlsystem of the present disclosure;

FIG. 3 is a flowchart illustrating an embodiment of a method of thepresent disclosure;

FIG. 4 is a schematic representation of a working environmentdemonstrating the operational use of embodiments of the presentdisclosure; and

FIGS. 5A & 5B illustrate the operational use of aspects of the presentdisclosure.

DETAILED DESCRIPTION

Systems and methods are provided for guiding an agricultural machine,(e.g. a harvester 10). Image data from one or more imaging sensors (e.g.cameras 29, LIDAR units, etc.) mounted, coupled or otherwise associatedwith the harvester 10 is obtained and used to detect objects (e.g.object ‘O’) within the environment of the harvester 10. Specifically,the image data is analysed using an object detection model to classifyat least one object within the environment of the harvester 10.Following classification, a cooperative machine is identified for theharvester 10, being a machine capable for performing an appropriatecooperative operational task with the harvester 10. As discussed herein,a cooperative machine for a harvester 10 may include a further harvesteroperating within the environment such that harvester 10 and the furtherharvester may work adjacent rows of the field. Operation of a guidancesystem of the harvester 10 can then be controlled in dependence on apositive identification of a cooperative agricultural machine. Asdiscussed herein, this may include controlling motion of the harvester10, e.g. through control over a propulsion, braking and/or steeringsystem of the harvester 10 for controlling motion of the harvester 10along an appropriate operational path for the cooperative operationaltask. The invention may extend to controlling further operablecomponents associated with the harvester 10, and may include a userinterface, e.g. display terminal 32 provided within an operator cab 30of the harvester 10 for displaying information indicative of theidentified cooperative machine, a determined operational path or thelike to an operator of the harvester 10, e.g. for providing guidanceinstructions to the operator.

Harvester

FIG. 1 illustrates an agricultural harvester 10, which includes achassis 12 supported and propelled along the ground by ground-engagingwheels 14. Although the harvester 10 is illustrated as being supportedand propelled on ground-engaging wheels 14, the harvester 10 can also besupported and propelled by full tracks or half tracks, as will beappreciated. A harvesting header 16 carried by the chassis 12 is used togather crop and to conduct the crop material to a feederhouse 18 andthen to a beater in the harvester 10. The beater guides the crop upwardto a threshing and separating system 20, which is configured to separategrain from material other than grain (MOG), and deliver the grain to agrain tank 28 carried by the harvester 10. The operation of theharvester 10 is controlled from an operator cab 30. A user interface 32is provided within the operator cab 30 for displaying or otherwiseproviding information to an operator of the harvester 10 data indicativeof the operation of the harvester 10 or other associated components,e.g. the header 16 or the guidance system as discussed herein.

The harvester 10 is additionally provided with an imaging sensor in theform of camera 29 mounted thereto and configured to obtain image datarepresentative of the working environment of the harvester 10, andanalysed in the manner discussed herein. In an extension of the presentdisclosure, the harvester 10 (or more generally the agriculturalmachine) may additionally or alternatively be provided with a differenttype of imaging sensor. For example, a transceiver-type sensor, such asa LIDAR sensor may be provided for imaging the environment of themachine. Advantageously, transceiver-type sensors may provide depthinformation for the environment, including a measure of a distancebetween the machine and the object. This can be utilised by the presentdisclosure, and in particular by the detection model for classifying oneor more objects within the environment of the machine.

Whilst discussed herein with reference to harvester 10, it will beapparent to the skilled reader that the systems and methods discussedherein may be applicable to any agricultural machine, including forageharvesters, self-propelled balers, tractors and tractor/implementcombinations, including others.

Control System

As discussed herein, a control system 100 is provided and configured tocontrol operation of one or more operable components (e.g. a guidancesystem 120, data server 130, and/or user interface 32) associated withthe harvester 10. In the illustrated embodiment the controllablecomponents include a guidance system 120 of the harvester forcontrolling operational parameters of the harvester 10, e.g. a forwardspeed or path, a data server 130 for storing information representativeof the classification outputs of the detection model, the determinedidentity of cooperative operational machines in the environment, and/ora determined operational path as determined in dependence on thecooperative machine identification as discussed herein; and the userinterface 32, here provided as a display terminal 32 provided within theoperator cab 30 of the harvester 10 as discussed, and utilised toprovide an indication of the operation of the control system 100, e.g. adetermined identity for a cooperative operational machine within theworking environment of the harvester 10 and/or a determined operationalpath for the harvester 10 for performance of the cooperative operationaltask.

FIG. 2 illustrates the control system 100 further. As shown, controlsystem 100 comprises a controller 102 having an electronic processor104, electronic input 106 electronic outputs 108, 112 and electronicinput/output 110. The processor 104 is operable to access a memory 114of the controller 102 and execute instructions stored therein to performthe steps and functionality of the present disclosure discussed herein,e.g. by controlling operation of operable components associated with theharvester 10 in dependence on a determined identity for one or moreobjects in the environment of the harvester 10, as determined by thecontrol system 100, optionally including controlling the user interface32 to display information indicative of determined identities and/or toreceive an operator input for initiating control actions based on thedetermined identities, through generation and output of one or morecontrol signals.

The processor 104 is operable to receive via input 106 which, in theillustrated embodiment, takes the form of input signals 105 receivedfrom a control unit associated with camera 29 associated with theharvester 10, image data representative of the working environment ofthe harvester 10. The processor 104 is configured to analyse the imagedata and determine therefrom a classification for one or more objects inthe environment in the manner discussed herein, i.e. using a trainedobject detection models for identifying the objects. The processor 104is further configured to utilise the determined classification toidentify a cooperative operational machine within the environment of theharvester 10.

As described above, the controller 102 includes an electronic output 108configured to output control signals 109 generated by the processor 104for controlling operation the guidance system 120 of the harvester 10.Specifically, in the illustrated embodiment, the processor 104 isoperable to generate, and the controller 102 operable then to output viaoutput 108, control signals 109 to local control unit of the guidancesystem 120 for controlling motion of the harvester 10 in dependence onthe identity of the cooperative machine as determined in the mannerdescribed herein. In practice, this may include control over a forwardspeed and/or operational path for the harvester 10 for performance ofthe cooperative task. The guidance system 120 may be controlled tocontrol operation of one or more of a steering system, braking systemand/or propulsion system of the harvester 10, for example, forcontrolling motion of the harvester 10 in the manner described herein.

Input/output 110 is operably connected to user interface 32. The controlsystem 100 is operable to control operation of the user interface 32,e.g. through output of control signals 111 in order to display data toan operator of the harvester 10 indicative of the identity of acooperative machine, as determined by processor 104. This can includesimply providing an indication to the operator of the determinedidentity(ies). This may include providing a graphical representation,optionally including image data from the camera 29, of the workingenvironment, object(s) located therein and any identities determined forthose objects as per the present disclosure. This may include presentinga representation of an operational path for the harvester 10 forperformance of the cooperative task. The input/output 110 isadditionally configured to receive input signals 111 from the userinterface 32 indicative of an operator input at the user interface 32,for example to provide the operator an ability to interact with thedata.

Output 112 is operably connected to a data server, preferably a remotedata server 130 for storing information indicative of the determinedidentities and/or determined operation of the guidance system. This maybe used, for example, by a remote user monitoring operation of multiplemachines within the working environment. Here, the image data from thecamera 29 can therefore be used to provide tracking data for multiplemachines to a remote server 130.

Method

FIG. 3 illustrates a method 200 in accordance with the presentdisclosure. Method 200 comprises, at step 202, receiving image data fromone or more image sensors (here camera 29) mounted to the harvester 10and operable, in use, to capture image data representative of theworking environment of the harvester 10. As shown herein, the camera 29is mounted on the operator cab 30 of the harvester 10 and is directedsuch that the field of view R of the camera 29 encompasses a regionahead of the harvester 10, in use. It will be appreciated that theinvention is not limited in this sense, and the camera 29 or indeed anyimaging sensor employed may be mounted with a field of view about anydirection of the working machine.

At step 204, the image data received, here from camera 29, is analysedutilising the object detection model to classify one or more objectswithin the environment of the harvester 10. The object detection modelscomprises a trained object detection algorithm, trained using one ormore training datasets having a plurality of images and known objects inthose images. The training dataset used for training of the detectionmodel may include multiple images annotated for multipleagriculture-specific classes. The annotated classes can include, forexample: individual agriculture machines and/or implements, including‘tractor’, ‘combine’, ‘trailer’, ‘combine header’, ‘baler’, or otherin-field objects, including ‘square bale’, ‘round bale’, ‘hedgerow’,etc.

In an embodiment, the object detection model is based on YOLOv5 trainedon this dataset. As would be understood, a YOLOv5 model is asingle-stage object detector and consist of a backbone network, neck anddetection head. The cross stage partial connections (CSP) backbonenetwork is used to extract features from the input image. The layers inthe neck of the network are inserted to collect feature maps fromdifferent stages. This enables the model to perform better whendetecting objects of various sizes and scales. In YOLOv5, PANet is usedas the neck network to obtain feature pyramids. The YOLO head appliesanchor boxes on features and generates final output vectors with classprobabilities, objectness scores and bounding boxes. The presentdisclosure utilises these output vectors to determine an identity for agiven object, and specifically to identify one or more cooperativemachines in the environment of the harvester 10.

At step 206, a cooperative agricultural machine is identified.Specifically, the classification(s) output by the object detection modelare analysed to determine which of one or more objects identified withinthe environment of the harvester 10 may represent a cooperativeagricultural machine for performing a cooperative operational task. Forharvester 10, and as discussed herein, this may include one or morefurther harvesting machines in the environment for working a pluralityof adjacent crop rows.

At step 208, the guidance system of the harvester 10 is then controlledbased on an identification of a cooperative agricultural machine. In anexample, a forward speed for the harvester 10 may be adjusted e.g.increased or reduced, in dependence on an identification of cooperativemachine to bring the harvester 10 into a cooperative working positionwith respect to the cooperative machine. As will be understood, thiscould include bringing the harvester 10 alongside the cooperativemachine, or to follow the cooperative machine, e.g. by matching theforward speed of the harvester 10 with the forward speed of theidentified cooperative machine. In practice, adjusting a forward speedof the harvester 10 includes control over a braking and/or propulsionsystem of the harvester 10, e.g. via the guidance system 120. This mayextend to controlling a steering system of the harvester 10, e.g. forcontrolling the harvester 10 to operate along an operational path forperformance of the cooperative operational task, which may includebringing the harvester 10 into alignment with the other machineinitially and/or retaining a cooperative alignment of the two (or more)machines during performance of the cooperative task.

Additionally or alternatively, step 208 may include output ofinformation relating to the determined identity(ies) to the remote dataserver 130, serving as a tracking system for tracking the position ofobjects, including other agricultural machines, within the workingenvironment.

Further, operable component(s) can, in embodiments, include control overthe user interface 32 in the manner discussed herein. For example,controlling the user interface 32 to display or otherwise present to theoperator an indication of determined identity(ies), and/or arepresentation of image data obtained by the camera 29 or otherapplicable imaging sensors. User interface 32 may be used to outputguidance instructions to an operator of the harvester 10 for performanceof the cooperative task with the identified cooperative machine.

Examples

FIG. 4 shows an example working environment F illustrating theoperational use of embodiments of the present disclosure discussedherein. Here, the present invention provides control of a guidancesystem of an agricultural machine in the form of tractor M0 and trailercombination. Here, one or more imaging sensors are provided mounted onthe tractor M0 and having an imaging region forward of the tractor M0for imaging the working environment. As discussed herein, the one ormore imaging sensors can include one or a combination of a camera, atransceiver sensor such as a LIDAR unit, amongst others.

In the illustrated example, the one or more imaging sensor associatedwith tractor M0 are used to obtain image data of the workingenvironment. Utilising the object detection model, classifications forvarious objects identified in the working environment can be determined.In this example, this includes a further tractor/trailer combination M2following a working path P2, an obstacle in the form of an electricitypole O1 fixed in position within the field F and a harvesting machine M1following a working path P1.

The classified objects are then analysed to identify a cooperativemachine for performing a cooperative operational task with thetractor/trailer combination M0. In this instance, this includesharvesting machine M1, and the cooperative task comprises an unloadingoperation for unloading crop material from a grain tank of theharvesting machine M1 to the trailer of the tractor/trailer combinationM0.

For performance of the cooperative task, an operational path PO isdetermined for the tractor/trailer combination M0 for bringing thetractor/trailer combination M0 into cooperative alignment with theharvesting machine. Specifically, this is a position where the traileris aligned with an unloading mechanism of the harvesting machine M1 forreceiving crop material therefrom. A guidance system of the tractor M0is then controlled for controlling movement of the tractor/trailercombination M0 along with determined operational path for performance ofthe cooperative task, e.g. through suitable control of a propulsion,braking and/or steering system of the tractor for guiding thetractor/trailer combination M0 along the path.

Aspects of the present disclosure are advantageously configured todistinguish between multiple objects within the working environment foridentification of a cooperative machine, and to control the guidancesystem accordingly for performance of a cooperative task with thatidentified machine.

In a variant, the agricultural machine may instead be a tractor with aworking implement suitably coupled thereto, e.g. at a rear hitch. Here,a suitable cooperative agricultural machine may include a furthertractor with a corresponding implement coupled thereto. An operationalpath for the tractor may be one which aligns the tractor and implementcombination with the corresponding cooperative machine such that thetractor and implement combination works an adjacent row to thecooperative machine. The implement may, in some instances, comprise aharrow, although the invention is equally applicable to other workingimplements.

In a further variant, the agricultural machine may be a baler, e.g. aself-baler or a baler unit suitably coupled and towed by a tractor. Acooperative machine may include a combine harvester performing aharvesting operation, and a suitable operational path may be one whichaligns the intake of the baler with a swath of residue material spreadby the combine during the harvesting operation, e.g. by positioning thebaler (or tractor/baler combination) directly behind and spaced from thecombine harvester.

In a further variant, the agricultural machine may comprise a harvestingmachine, such as combine harvester 10. A cooperative agriculturalmachine may be a corresponding harvesting machine, and a suitableoperational path may be one which aligns the harvester 10 with thecorresponding cooperative harvesting machine such that the harvester 10works in parallel with the cooperative harvesting machine—e.g. in thenext crop row.

Other variants may be equally applicable, and the present disclosure isnot limited in this sense.

As discussed, the present disclosure extends to control over a userinterface, e.g. a display terminal 32 provided within an operator cab 30of the harvester 10 (or like components of other agricultural machines.FIGS. 5A and 5B illustrate example representations that may be providedby a suitable user interface (e.g. display terminal 32) upon successfulidentification of a cooperative machine (or indeed classification ofother objects within the environment). For example, FIG. 5A represents arepresentation of image data obtained by a camera 29′ provided on anagricultural machine in the form of a tractor 10′. Here an object ‘O’has been identified in the manner discussed herein, and an appropriatebounding box B and label L has been provided to highlight the positionand identity of the object—here another tractor—to an operator of thetractor 10′. This may be provided for purely informational purposes tothe operator of the tractor 10′, or could provide means for the operatorto interact with the interface—e.g. to select a cooperative machine onthe interface, or to provide further instructions to the system—e.g.initiating performance a cooperative task. FIG. 5B shows arepresentation of image data obtained by a LIDAR unit also provided onthe tractor 10′. Again, the object ‘O’ is identified and highlighted inthe image data from the sensor—LIDAR unit. The LIDAR unit may be used toprovide depth information which is used in conjunction with the imagedata from camera 29′ for identifying the object O. Depth information mayalso be utilised, as will be appreciated, for control over the relativedistances between the agricultural machine and a cooperative machine,e.g. whilst performing the cooperative operational task.

General

Any process descriptions or blocks in flow diagrams should be understoodas representing modules, segments, or portions of code which include oneor more executable instructions for implementing specific logicalfunctions or steps in the process, and alternate implementations areincluded within the scope of the embodiments in which functions may beexecuted out of order from that shown or discussed, includingsubstantially concurrently or in reverse order, depending on thefunctionality involved, as would be understood by those reasonablyskilled in the art of the present disclosure.

It will be appreciated that embodiments of the present invention can berealized in the form of hardware, software or a combination of hardwareand software. Any such software may be stored in the form of volatile ornon-volatile storage such as, for example, a storage device like a ROM,whether erasable or rewritable or not, or in the form of memory such as,for example, RAM, memory chips, device or integrated circuits or on anoptically or magnetically readable medium such as, for example, a CD,DVD, magnetic disk or magnetic tape. It will be appreciated that thestorage devices and storage media are embodiments of machine-readablestorage that are suitable for storing a program or programs that, whenexecuted, implement embodiments of the present invention. Accordingly,embodiments provide a program comprising code for implementing a systemor method as set out herein and a machine readable storage storing sucha program. Still further, embodiments of the present invention may beconveyed electronically via any medium such as a communication signalcarried over a wired or wireless connection and embodiments suitablyencompass the same.

All references cited herein are incorporated herein in their entireties.If there is a conflict between definitions herein and in an incorporatedreference, the definition herein shall control.

What is claimed is:
 1. A control system for a guidance system of anagricultural machine, the control system comprising one or morecontrollers, and being configured to: receive image data from one ormore imaging sensors associated with the agricultural machine; analysethe image data utilising a detection model to classify at least oneobject within the environment of the agricultural machine; identify,from the at least one classified object, a cooperative machine forperforming a cooperative operational task with the agricultural machine;and generate and output one or more control signals for controllingoperation of the guidance system in dependence on the identifiedcooperative machine.
 2. A control system as claimed in claim 1, operableto control the guidance system in dependence on the identity and/orposition of the identified cooperative machine.
 3. A control system ofclaim 1, wherein the guidance system comprises a user interface; and thecontrol system is configured to control operation of the user interfacefor outputting one or more indicators indicative of the identity and/orposition of the cooperative machine.
 4. A control system of claim 3,configured to control operation of the user interface of the guidancesystem for outputting one or more instructions for an operator of theagricultural machine relating to movement of the agricultural machinewith respect to the cooperative machine.
 5. A control system of claim 1,configured to determine an operational path for the agricultural machinefor performing the cooperative operational task.
 6. A control system ofclaim 5, configured to generate and output one or more control signalsfor controlling operation of the guidance system of the agriculturalmachine for controlling motion of the agricultural machine along thedetermined operational path.
 7. A control system of claim 6, configuredto control: a propulsion system of the agricultural machine forcontrolling a forward speed of the agricultural machine; and/or asteering system of the agricultural machine for controlling steering ofthe machine along or with respect to the determined operational path orwith respect to the cooperative machine or one or more componentsthereof.
 8. A control system of claim 5, wherein: the operational pathcomprises a suggested or determined path for moving the agriculturalmachine to an operational position with respect to the cooperativemachine; and/or the operational path comprises a suggested or determinedpath for moving the agricultural machine along a path to perform thecooperative operational task.
 9. A control system of claim 1, configuredto utilise the detection model for determining a classification for eachof a plurality of objects within the image data.
 10. A control system ofclaim 9, configured to: determine, from the multiple classifications,one or more candidate cooperative machines; select one of the one ormore candidate cooperative machines; and control the guidance system ofthe agricultural machine in dependence thereon.
 11. A control system ofclaim 1, wherein the detection model comprises a machine-learned modeltrained on one or more training datasets with known objects withrespective classifications.
 12. A control system of claim 1, wherein theclassification output by the detection model comprises a bounding boxoverlaid onto the image data at the location of the object as determinedby the respective model; and wherein the position of the bounding boxwithin the image data is utilised by the control system for controllingthe guidance system.
 13. A control system as claimed in claim 1, whereinthe agricultural machine comprises a tractor with a trailer suitablycoupled thereto; and the control system is configured to: identify acooperative agricultural machine within the environment of the tractoras a combine harvester performing a harvesting operation; and controlthe guidance system in accordance with an operational path which alignsthe trailer with an unloading mechanism of the combine for receivingcrop material from the combine in the trailer of the tractor-trailercombination.
 14. A control system of claim 1, wherein the agriculturalmachine comprises a tractor with an implement suitably coupled thereto;and the control system is configured to: identify a cooperativeagricultural machine within the environment as a tractor with acorresponding implement coupled thereto; and control the guidance systemin accordance with an operational path which aligns the tractor andimplement combination with the corresponding cooperative machine suchthat the tractor and implement combination works an adjacent row to thecooperative machine.
 15. A control system of claim 1, wherein theagricultural machine comprises a baler; and the control system isconfigured to: identify a cooperative agricultural machine within theenvironment of the tractor as a combine harvester performing aharvesting operation; and control the guidance system in accordance withan operational path which aligns the intake of the baler with a swath ofresidue material spread by the combine during the harvesting operation.16. A control system of claim 1, wherein the agricultural machinecomprises a harvesting machine; and the control system is configured to:identify a cooperative agricultural machine within the environment as acorresponding harvesting machine; and control the guidance system inaccordance with an operational path which aligns the harvesting machinewith the corresponding cooperative harvesting machine such that theharvesting machine works in parallel with the cooperative harvestingmachine.
 17. A guidance system for an agricultural machine, comprisingone or more imaging sensors; and wherein the system further comprisesand/or is controllable by the control system of claim
 1. 18. Anagricultural machine comprising a control system of claim
 1. 19. Amethod for guiding an agricultural machine, comprising: receiving imagedata from one or more imaging sensors associated with the agriculturalmachine; analysing the image data utilising a detection model toclassify at least one object within the environment of the agriculturalmachine; identifying, from the at least one classified object, acooperative machine for performing a cooperative operational task withthe agricultural machine; and controlling operation of the guidancesystem in dependence on the identified cooperative machine.